{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Bayesian deep learning for cosmology and gravitational waves\n", "\n", "Tom Charnock\n", "\n", "Institut d'Astrophysique de Paris\n", "


\n", "\n", "\n", "Slides available at presentations.charnock.fr/BDL\n", "\n", "
\n", "
\n", " \"Sorbonne\n", "
\n", "
\n", " \"ANR\"\n", "
\n", "
\n", " \"IAP\"\n", "
\n", "
\n", " \"CNRS\"\n", "
\n", "
\n", " \"Aquila\"\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "#### What is a neural network?\n", "\n", "#### What is Bayesian statistics?\n", "\n", "#### Why there is a problem with *traditionally* trained neural networks\n", "\n", "#### What is a neural network formulated as a Bayesian model?\n", "\n", "#### How can Bayesian neural networks be implemented?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What is a neural network?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", "

Fully connected

\n", "
\n", " \n", "
\n", "
\n", "
\n", "

Convolutional

\n", "

\n", " \n", "
\n", "
\n", "\n", "
 
\n", " \n", "
\n", "
\n", "

Autoregressive flows

\n", " \n", "
\n", "
\n", "

Recurrent

\n", "
\n", " \n", "
\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# A neural network is just a parameterised function\n", "

\n", "$$\\Large\\mathbb{NN}(\\boldsymbol{\\omega},\\boldsymbol{\\alpha}) : {\\bf x} \\to {\\bf y}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "### We can fit the neural network to a desired function using data\n", "
\n", "$$\\Large\\mathcal{D} = \\{{\\bf x},{\\bf y}\\}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Some examples of possible ${\\bf x}$ and ${\\bf y}$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Mapping inputs to ouputs: Classification, regression, etc.\n", "

\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Learning latent representations\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Predicting posterior distribution of targets\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Bayesian statistics" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Statistical models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Aleatoric uncertainty\n", "\n", "##### Uncertainty due to the true underlying statistical model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Epistemic uncertainty\n", "\n", "##### Uncertainty which arises due to the lack of knowledge about the true statistical model" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Prior belief\n", "##### Quantifying what we don't know...\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### Characterising the epistemic uncertainty
(although it is not the epistemic uncertainty)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Creating a statistical model\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Joint probability of data and parameters\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# The likelihood and the posterior\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Likelihood-free (simulation based) inference\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Bayes theorem\n", "\n", "##### Epistemic uncertainty in the values of the parameter can be reduced under the evidence of observed data\n", "

\n", "$$\\Large\\mathcal{P}(\\theta|\\mathcal{D},\\psi)=\\frac{\\mathcal{L}(\\mathcal{D}|\\theta,\\psi)\\mathscr{p}(\\theta|\\psi)}{\\mathscr{p}(\\mathcal{D}|\\psi)}$$\n", "

" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "notes" } }, "source": [ "# Reduce total epistemic uncertainty using\n", "$$\\Huge\\mathcal{P}(\\theta,\\psi|\\mathcal{D})=\\frac{\\mathcal{L}(\\mathcal{D}|\\theta,\\psi)\\mathscr{p}(\\theta|\\psi)\\mathscr{p}(\\psi)}{\\mathscr{p}(\\mathcal{D})}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Neural networks formulated as Bayesian models" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## How likely is any predicted output?\n", "
\n", "
\n", "$$\\mathcal{P}(\\mathscr{y}|{\\bf x})=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### A neural network is deterministic\n", "
\n", "
\n", " $$\\mathcal{L}({\\bf y}|{\\bf x},\\omega,\\alpha)=\\delta(\\bf{y}-\\mathscr{y})$$\n", "
\n", "
\n", " $$\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)=1$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# How likely is any predicted output?\n", "

\n", "$$\\mathcal{P}(\\mathscr{y}|{\\bf x})=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Why neural networks do not work..." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Choose a neural network\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "$$\\begin{align*}\n", " \\mathcal{P}(\\mathscr{y}|{\\bf x})&=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)\\\\\n", " &=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\delta(\\alpha-\\alpha^*)\\\\\n", " &=\\int_\\Omega d\\omega\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha^*)\\mathscr{p}(\\omega|\\alpha^*)\\\\\n", "\\end{align*}$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Maximum likelihood estimates of the weights\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "$$\\omega_\\textrm{MLE}=\\underset{\\omega\\in\\Omega}{\\textrm{argmax}}\\,\\mathcal{L}(\\mathcal{D}|\\omega,\\alpha^*)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Deterministic neural network\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "$$\\begin{align*}\n", " \\mathcal{P}(\\mathscr{y}|{\\bf x})&=\\int_\\Omega d\\omega\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha^*)\\mathscr{p}(\\omega|\\alpha^*)\\\\\n", " &=\\int_\\Omega d\\omega\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha^*)\\delta(\\omega-\\omega_\\textrm{MLE})\\\\\n", " &=\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega_\\textrm{MLE},\\alpha^*)\\\\\n", " &=1\n", "\\end{align*}$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### No characterisation of the epistemic uncertainty" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Neural density estimators" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Mixture density networks\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Normalising flows\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# But...\n", "\n", "$$\\begin{align*}\n", " \\mathcal{P}(\\mathcal{P}(\\mathscr{y}|{\\bf x})|{\\bf x})&=\\int_\\Omega d\\omega\\,\\mathcal{L}(\\mathcal{P}(\\mathscr{y}|{\\bf x})|{\\bf x},\\omega,\\alpha^*)\\mathscr{p}(\\omega|\\alpha^*)\\\\\n", " &=\\int_\\Omega d\\omega\\,\\mathcal{L}(\\mathcal{P}(\\mathscr{y}|{\\bf x})|{\\bf x},\\omega,\\alpha^*)\\delta(\\omega-\\omega_\\textrm{MLE})\\\\\n", " &=\\mathcal{L}(\\mathcal{P}(\\mathscr{y}|{\\bf x})|{\\bf x},\\omega_\\textrm{MLE},\\alpha^*)\\\\\n", " &=1\n", "\\end{align*}$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Bayesian neural networks" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "$$\\mathcal{P}(\\mathscr{y}|{\\bf x})=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\,\\mathcal{L}(\\mathscr{y}|{\\bf x},\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)$$\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Networks not used in a Bayesian context, but working with distributions\n", "\n", "##### Model $\\mathscr{p}(\\omega|\\alpha)$ with some distribution\n", "\n", "##### Maybe model $\\mathscr{p}(\\alpha)$ with some distribution (although probably not)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Networks used in a Bayesian context" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "

\n", "\n", "$$\\large\\mathcal{P}(\\omega,\\alpha|\\mathcal{D})=\\frac{\\mathcal{L}(\\mathcal{D}|\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)}{\\mathscr{p}(\\mathcal{D})}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Almost intractable posterior of network parameters\n", "

\n", "$$\\mathcal{P}(\\omega,\\alpha|\\mathcal{D})$$\n", "\n", "### Almost certainly intractable evidence \n", "

\n", "$${\\mathscr{p}(\\mathcal{D})=\\int_\\Omega\\int_\\mathcal{A}d\\omega d\\alpha\\mathcal{L}(\\mathcal{D}|\\omega,\\alpha)\\mathscr{p}(\\omega|\\alpha)\\mathscr{p}(\\alpha)}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Practical implementations" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Numerically approximate methods using the exact distribution
Markov chain Monte Carlo methods" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Markov chains\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "\n", "Stationary distribution, $\\pi(\\omega)\\approx\\mathcal{P}(\\omega|\\mathcal{D},\\alpha^*)$,
ensured by detailed balance

\n", "$$\\pi(\\omega^i)t(\\omega^{i+1}|\\omega^i)=\\pi(\\omega^{i+1})t(\\omega^i|\\omega^{i+1})$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Dropping the evidence\n", "$$\\begin{align*}\n", " \\frac{t(\\omega^{i+1}|\\omega^i)}{t(\\omega^i|\\omega^{i+1})}&=\\frac{\\mathcal{P}(\\omega^{i+1}|\\mathcal{D},\\alpha^*)}{\\mathcal{P}(\\omega^i|\\mathcal{D},\\alpha^*)}\\\\\\\\\n", " &=\\frac{\\mathcal{L}(\\mathcal{D}|\\omega^{i+1},\\alpha^*)\\mathscr{p}(\\omega^{i+1}|\\alpha^*)/\\mathscr{p}(\\mathcal{D}|\\alpha^*)}{\\mathcal{L}(\\mathcal{D}|\\omega^i,\\alpha^*)\\mathscr{p}(\\omega^i|\\alpha^*)/\\mathscr{p}(\\mathcal{D}|\\alpha^*)}\\\\\\\\\n", " &=\\frac{\\mathcal{L}(\\mathcal{D}|\\omega^{i+1},\\alpha^*)\\mathscr{p}(\\omega^{i+1}|\\alpha^*)}{\\mathcal{L}(\\mathcal{D}|\\omega^i,\\alpha^*)\\mathscr{p}(\\omega^i|\\alpha^*)}\\\\\\\\\n", " &=\\frac{\\varrho(\\omega^{i+1})}{\\varrho(\\omega^i)}\n", "\\end{align*}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## ~~Metropolis~~ Rosenbluth-Hastings\n", "\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Problems with Metropolis-Hastings\n", "##### High rejection rate\n", "##### Extremely expensive in many dimensions (random walk)\n", "\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Hamiltonian Monte Carlo\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Mass matrix describes correlations - these are not known *a priori*" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Quasi-Newonian HMC\n", "\n", "##### Use the approximate Hessian, ${\\bf B}$, for additional geometric information\n", "\n", "
\n", "$$\\begin{align*}\n", " \\dot{\\boldsymbol{\\omega}} &={\\bf B}{\\bf M}^{-1}\\boldsymbol{p} \\\\\n", " \\dot{\\boldsymbol{p}} &={\\bf B}\\nabla\\ln\\varrho(\\boldsymbol{\\omega})\n", "\\end{align*}$$\n", "\n", "##### Rescales the mass matrix so that steps are taken in a useful direction" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Sampling the exact posterior of neural network parameters is possible...\n", "\n", "##### But it is hyper-expensive and hyper-high-dimensional\n", "\n", "##### There are only a small handfull of actual applications (bit.ly/NeuralBiasModel)
and most others are proof of concept" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Exact treatment of approximate distributions
Variational inference" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Approximate the posterior distribution of weights with analytic approximations\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Analytic (or automatic) parameterised distributions\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Normalising flows\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Kullback–Leibler (KL) divergence\n", "\n", "##### Amount of information lost by approximating $\\mathcal{P}(\\omega|\\mathcal{D})$ with $\\mathcal{Q}(\\omega)$\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "$$\\mathbb{KL}(\\mathcal{P}||\\mathcal{Q})=\\int_\\Omega d\\omega\\, \\mathcal{P}(\\omega|\\mathcal{D})\\ln\\frac{\\mathcal{P}(\\omega|\\mathcal{D})}{\\mathcal{Q}(\\omega)}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### How variational inference works\n", "\n", "For variational inference we normally consider\n", "

\n", "$$\\mathbb{KL}(\\mathcal{Q}||\\mathcal{P})=\\int_\\Omega d\\omega\\,\\mathcal{Q}(\\omega)\\ln\\frac{\\mathcal{Q}(\\omega)}{\\mathcal{P}(\\omega|\\mathcal{D})}$$\n", "
\n", "since the expectation value of $\\mathcal{P}(\\omega|\\mathcal{D})$ is intractable.\n", "

\n", "Minimising the reverse KL divergence is equivalent to expectation-maximisation." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Evidence Lower BOund (ELBO)\n", "\n", "How do we deal with the intractable posterior $\\mathcal{P}(\\omega|\\mathcal{D})$?\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", "$$\\begin{align*}\n", "\\small\\mathbb{KL}(\\mathcal{Q}||\\mathcal{P}) &\\small= \\int_\\Omega d\\omega\\,\\mathcal{Q}(\\omega)\\ln\\frac{\\mathcal{Q}(\\omega)}{\\mathcal{P}(\\omega|\\mathcal{D})}\\\\\n", "&\\small=\\mathbb{KL}(\\mathcal{Q}||\\varrho)+\\ln\\mathscr{p}(\\mathcal{D})\\\\\n", "\\end{align*}$$\n", "
\n", "
\n", " \n", " \n", "The unnormalised posterior, $\\varrho(\\omega|\\mathcal{D})$, can be evaluated\n", "\n", "$$\\mathbb{KL}(\\mathcal{Q}||\\varrho)=\\mathbb{KL}(\\mathcal{Q}||\\mathcal{P})-\\ln\\mathscr{p}(\\mathcal{D})$$\n", "
\n", "Minimising $\\mathbb{KL}(\\mathcal{Q}||\\varrho)$ (maximising the ELBO) also minimises $\\mathbb{KL}(\\mathcal{Q}||\\mathcal{P})$\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Note, this is NOT Bayesian" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Mean field approximation\n", "
\n", "
\n", "
\n", " \n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Variational Bayes\n", "\n", "##### Model the posterior of weights with some parameterised distribution\n", "\n", "##### Update the parameters with by maximising the ELBO." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Which distributions?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Neural network parameters are not identifiable...\n", "\n", "- Shallow local maxima\n", "\n", "- Highly degenerate\n", "\n", "- Super multimodal\n", "\n", "##### We need something to model this!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## MCDropout (the worst *usable* choice?)\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### Weird sampling of combinations of neurons, no parameters learned" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# DropConnect\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### At least its actually sampling weights, no parameters learned" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Variational Dropout\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### Error added to weights, not weight posterior, no parameters learned" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Reparameterisation trick\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### Actual characterised posterior distribution for weights, only Gaussian" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Flowed reparameterisation\n", "\n", "
\n", "
\n", " \n", "
\n", "
\n", "\n", "##### Actual characterised posterior distribution for weights" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Variational Inference is tractable\n", "\n", "##### It can run in a reasonable time" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Variational Bayes isn't yet very Bayesian...\n", "\n", "##### People get confused between Bayesian and neural networks using distributions\n", "\n", "##### We should use the posterior distribution of weights and the form of the distribution\n", "\n", "##### Instead we choose a (poor) choice of representation of the distribution and fit that\n", "\n", "##### Epistemic uncertainty is not reduced (and may be increased or biased)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What do we actually want?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "##### The posterior predictive distribution, $\\mathcal{P}(\\mathscr{y}|{\\bf x})$?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Likelihood-free inference" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "##### Use a (well) trained network to find __summaries__\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Generate simulations drawn from prior and summarise\n", "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Make a kernel (or neural) density estimate (or use ABC) of the summary-parameter pairs\n", "
\n", "\n", "
A slice through at the summarised observation is the approximate posterior
" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "# Pros and cons\n", "\n", "##### Easy to use - can stop worrying about Bayesian neural networks\n", "\n", "##### Guaranteed posterior (of summaries)\n", "\n", "##### Density estimator can be precomputed and evaluated quickly\n", "\n", "##### There is a choice of density estimator\n", "\n", "##### Depends on the number of simulations (not different than other methods)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### So... what have we learned?\n", "\n", "##### Neural networks can be formulated as statistical models\n", "\n", "##### Epistemic uncertainties can (in principle) be reduced via Bayes\n", "\n", "##### Bayesian neural networks mean different things to different people\n", "\n", "##### Actual (as in sampled) neural networks are REALLY expensive\n", "\n", "##### Variational approximation to weight posterior exist, aren't Bayesian\n", "\n", "##### Likelihood-free inference could provide a non-Bayesian neural network Bayesian method with neural networks" ] } ], "metadata": { "celltoolbar": "Diaporama", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 2 }